{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# STAT40800 Midterm Assignment\n",
    "## Exploratory data analysis of the Billboard Hot 100 Chart\n",
    "\n",
    "For this assignment you will perform an exploratory data analysis (EDA) for historic data from Billboard Hot 100 Chart. The Billboard Hot 100 is the music industry standard record chart in the United States for songs, published weekly by Billboard magazine. Chart rankings are based on sales, radio play, and online streaming in the United States.\n",
    "\n",
    "Every week, Billboard releases *The Hot 100* chart of songs that were trending on sales and airplay for that week. The dataset you will analyse is a collection of all *The Hot 100* charts released since its inception in 1958. The song with rank 1 is the most popular song that week.\n",
    "\n",
    "Marks will be awarded for complete and correct answers to all 10 questions. For full marks, you must show clearly all steps and computations in your answers. **You must submit your completed Jupyter notebook in .ipynb and PDF format**. To create a PDF of your notebook select *File* $\\rightarrow$ *Download as* $\\rightarrow$ *PDF via LaTeX (.pdf)*.\n",
    "\n",
    "This assignment is **due on Oct 29th at 6pm**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in necessary packages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Load the `charts.csv` dataset into Python as a pandas DataFrame. Include the arguments `parse_dates=True` in the read_csv function to convert the date strings into datetime objects. Describe the data. Comment on the size of the dataset and the type of data in each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Write code to answer the following questions:\n",
    "    1. How many different dates are included in the data set?\n",
    "    * How many different artists featured in the charts since its inception in 1958?\n",
    "    * Determine the top 10 artists that feature in the charts the most amount of times?\n",
    "    * What is the maximum length of time a song featured in the charts? What was this song and who was the artist?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Determing how many missing values there are in each column of the dataset. Can you think of a reason why why these values are missing? Replace any missing values in the `last-week` column with the integer 101. Is this a reasonable value to replace the missing values with?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. For each entry compute the difference between its rank last week and its current rank. Add the change in ranking as an additional column to the DataFrame. Create a numerical summary of change in ranking (mean, standard deviation, minimum, maximum, etc.) and interpret your results.\n",
    "\n",
    "    **Note:** If the song is higher up the charts this week, the change in ranking should be positive. While if the song falls down the charts, the change in ranking should be negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Create a scatter plot of the current rank versus the change in ranking from the previous week. Describe and interpret the plot. Compute the correlation coefficient between the current rank and the change in ranking. Does is agree with your what you found in your plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Songs will typically appear in the charts for a number of weeks, as such, there are a number of duplicate entries. Delete all duplicate entries, keeping only the most recent entries of a particular song by a particular artist in the charts. How many unique songs (with unique artists) have appeared in the charts since its inception in 1958? Repeat Q2C for the new DataFrame, with unique entries only. Are the top 10 entries the same? If not, discuss why they may have changed.\n",
    "\n",
    "    You may find the pandas method `drop_duplicates` useful. See the [pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop_duplicates.html) for details on how to apply this method to a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7. Recompute the summary statistics for the change in ranking from the previous week and recreate the scatter plot of the current rank versus the change in ranking for the new DataFrame. Discuss and interpret any changes in these statistics and the plot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8. Using the new DataFrame, with unique entries only, create a histogram for the length of time each entry spent in the charts. Describe and interpret your plot. Compute the summary statistics for the length of time a song spends in the charts. Discuss these statistics in relation to your histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9. Using the new DataFrame, with unique entries only, create a histogram for the peak rank. Describe and interpret your plot. Compute the summary statistics for peak rank. Discuss these statistics in relation to your histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10. Write a short summary on what you have learnt about the Billboard Hot 100 Chart from this analysis. (5-10 sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}